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Introduction 


To  properly  understand  science  and  technology  (S&T)  leadership  and  decisionmaking 
in  government  laboratories,  the  Center  for  Technology  and  National  Security  Policy 
Homeland  Security  team  conducted  a  study  on  practical  approaches  to  S&T  risk- 
informed  decisionmaking  and  metrics  for  program  selection.  The  study  was  conducted 
for  the  Director  of  Research  of  the  Science  and  Technology  Directorate  at  the 
Department  of  Homeland  Security  (DHS).  The  results  are  applicable  to  other  S&T 
organizations. 

The  S&T  Directorate  is  the  primary  research  and  development  ann  of  DHS.  The 
directorate's  mission  is  to  improve  homeland  security  by  providing  customers  state-of- 
the-art  technology  that  helps  them  achieve  their  missions.  S&T  customers  include  the 
operating  components  of  DHS,  state,  local,  tribal,  and  territorial  emergency  responders 
and  officials.  The  Directorate  has  three  portfolios  (research,  innovation  and  transition) 
that  address  basic  research  through  advanced  technology  development  and  transition  - 
spanning  six  primary  divisions  (Borders  and  Maritime  Security;  Chemical  and 
Biological;  Command,  Control  and  Interoperability;  Explosives;  Human  Factors 
Behavioral  Sciences;  Infrastructure  and  Geophysical)  that  address  critical  homeland 
security  needs. 

The  Director  of  Research  is  one  of  three  functional  portfolio  managers  reporting  to 
the  Undersecretary  for  Science  and  Technology.  The  others  are  the  Director  of 
Innovation  and  the  Director  of  Transition.  These  portfolio  managers  provide  consistency 
in  their  respective  areas  of  responsibility  across  the  spectrum  of  the  DHS  S&T 
investment.  Currently,  four  DHS  laboratories  report  to  the  Director  of  Research,  while 
two  report  to  the  Director  of  Test  &  Evaluation  and  Standards.  There  are  six  divisions 
that  also  report  to  the  Office  of  the  Undersecretary,  where  the  majority  of  the 
programmatic  resourcing  and  execution  originates.  Most  of  the  DHS  laboratory  research 
budget  funnels  through  one  of  the  six  divisions  within  S&T.  DHS  internal  management 
and  regulation  continue  to  evolve  as  the  department  meets  new  national  security 
challenges  that  demand  solutions  backed  by  scientific  and  technological  approaches. 

The  goals  of  this  study  are:  first,  to  apply  leadership  lessons  learned  from  the 
Department  of  Defense  laboratory  management  experience  to  the  needs  of  DHS;  second, 
to  recommend  best  practices  for  leadership  in  the  S&T  Directorate;  and  third,  to  respond 
to  DHS-requested  areas  of  evaluation.1  This  paper  synthesizes  the  results  of  this  study. 
Initially,  the  CTNSP  team  concentrated  on  the  need  for  practical  approaches  that  the  DHS 
leadership  could  use  to  make  critical  risk-informed  decisions  and  developed  metrics  for 
the  selection  of  the  most  beneficial  programs  and  projects. 

The  subject  of  risk-informed  decisionmaking  has  received  considerable  attention  in 
the  literature  due  to  its  broad  application  to  a  number  of  disciplines.  It  is  basically  “a 
process  that  organizes  information  about  the  possibility  for  one  or  more  unwanted 


April  7,  2009  meeting  between  CTNSP  Homeland  Security  team  and  Department  of  Homeland  Security 
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outcomes  into  a  broad,  orderly  structure  that  helps  decisionmakers  make  more  informed 
management  choices”.2  The  major  components  of  the  process  are3: 

1 .  Establish  the  decision  structure 

2.  Perform  the  risk  assessment 

3.  Apply  the  results  to  risk  management  decisionmaking 

4.  Monitor  effectiveness  through  impact  assessment 

5.  Facilitate  risk  communication 

The  co-authors  of  this  paper  formed  the  team  which  conducted  this  study.  Brief 
biographies  of  each  member  are  at  the  front  of  this  paper.  The  team  brought  to  the  table 
decades  of  combined  experience  in  managing  civilian  and  military  laboratories  and 
research  centers. 

This  paper  discusses  risk  and  impact  areas  in  relation  to  decisionmaking  and  the 
development  of  metrics  or  a  figure  of  merit  for  decisionmaking.  The  metrics  are  then 
applied  to  three  examples  of  interest  to  the  Army,  Air  Force,  and  DHS.  The  decision 
process  utilizing  the  figure  of  merit  is  described  for  S&T  program  selection  and  funding. 
Various  styles  of  decisionmaking  used  for  the  management  of  DOD  laboratories  is  briefly 
described  and  the  differences  between  management  and  leadership  are  highlighted. 


2  Principles  of  Risk-Based  Decision  Making,  ABS  Consulting,  Government  Institutes,  Rockville, 
Maryland,  2001. 

3  Ibid 
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Methodologies  for  Risk-Informed  Decisionmaking 

Several  methodologies  for  risk-informed  decisionmaking  were  examined  for  the 
purpose  of  identifying  metrics  for  program  evaluation  and  selection.  A  summary  of  these 
methodologies  is  highlighted  here  and  in  Appendix  A.  The  team  performed  research  and 
analysis  of  relevant  material  dealing  with  risk-informed  decisions  in  scientific  institutions 
and  organizations  working  with  issues  related  to  DHS.  This  material  included  books  and 
periodicals  about  managing  scientific  institutions;  newspaper  and  magazine  articles  about 
best  practices  in  managing  scientific  organizations  and  making  decisions  in  government¬ 
funding  institutions;  and  analyses  of  best  practices  in  managing  government-funded 
scientific  laboratories  over  the  past  several  decades.  Additionally,  the  team  reviewed 
available  infonnation  on  privately  run  companies  and  institutions  working  on  scientific 
and  technological  research.  Appendix  A  outlines  the  performance  evaluation  and  metrics 
for  program  selection  by  the  Army  Research  Laboratory.  This  methodology  was  used  to 
evaluate  the  health  and  performance  of  the  Laboratory  and  was  not  used  in  this  paper. 

Another  technique  was  used  by  the  Goldense  Group  in  measuring  overall  research 
and  development  (R&D)  program  effectiveness4.  One  way  to  measure  overall  R&D 
program  effectiveness  is  as  follows: 


ROII  = _ (Cumulative  N-year  Pro  ft  from  New  Products) 

(Cumulative  N-year  Expenditure  on  New  Product  Development) 

wherein:  ROII  =  return  on  innovation  investment 

N  =  a  given  number  of  years  after  the  investment  is  made 


The  numerator  is  sometimes  referred  to  as  “profit  before  tax.”  The  above  equation  has 
no  time  value  term.  By  using  the  net  present  value  (NPV)  for  both  the  numerator  and 
denominator,  one  can  take  time  factors  into  account.  NPV  can  be  affected  by  inflation 
and  other  changes  in  the  cost  of  raw  materials,  salaries,  and  sale  price  of  the  new  product. 
NPV  allows  comparison  of  projects  with  different  time  frames  and  different  investment 
requirements. 

This  proposal  offered  another  key  metric,  the  new  product  development  (NPD) 
success  rate.  This  is  defined  as  the  fraction  of  NPD  projects  entering  the  commercial 
development  stage  that  become  commercial  successes,  meeting  or  exceeding  financial 
objectives.  The  average  success  rate  for  U.S.  industry  is  60.2%.  This  methodology  was 
not  used  here  since  it  is  intended  to  identify  program  effectiveness  versus  decisionmaking 
in  selecting  programs. 

Another  methodology  responds  to  the  question  of  whether  risk  management  makes  a 
difference;  this  infonnation  is  based  on  a  paper  by  the  Association  of  Reported  Project 


4  Running  Your  Lab  Like  a  Business,  by  John  Borchardt,  From  "Leadership  and  Staffing"  section  of  the 
LAB  MANAGER  magazine  (P.O.  Box  216,  478  Bay  Street  Midland,  ON,  Canada,  L4R  1K9),  Published 
7/21/2008 


3 


Risk  Management  Practices  and  Project  Success/Project  Management  Institute.5  The 
overall  question  answered  by  this  paper  was:  does  risk  management  make  a  difference? 
That  is,  do  organizations  that  employ  formal  risk  management  practices  outperfonn  those 
that  do  not?  Data  collected  from  175  web-survey  respondents  and  12  selective  telephone 
interviewees  from  the  Risk  Management  Specific  Interest  Group  of  the  Project 
Management  Institute  answer  this  question  in  the  affirmative,  and  the  data  analysis 
supports  the  positive  influence  of  project  risk  management.  Several  detailed  conclusions 
are  presented  and  explained. 

In  the  course  of  this  research  we  found  that  adequate  resource  allocation  and  staff 
training  for  project  risk  management  are  less  pronounced  than  risk  visibility  in 
organizational  policymaking,  and  that  fewer  than  half  of  the  respondents  (45%)  use 
quantitative  tools  on  an  “Almost  Always”  basis  to  develop  contingency  costs.  Even  fewer 
use  them  to  estimate  contingency  times  (38%)  or  to  select  projects  (7%).  In  contrast, 
analysis  of  the  data  supports  generally  positive  influence  of  project  risk  management.  In 
particular,  the  following  four  general  conclusions  can  be  made: 

1.  The  more  sensitive  senior  management  is  (perceived  to  be)  to  project  risk 
management,  the  more  frequent  is  the  use  of  various  project  risk  management  practices. 
For  example,  37%  of  respondents  from  organizations  with  a  stated  risk  management 
policy  report  using  qualitative  risk  tools  “Almost  Always”  versus  only  8%  of  the 
respondents  from  organizations  without  such  a  policy. 

2.  The  more  that  senior  managers  provide  adequate  resources  for  risk  management 
processes,  the  more  frequent  is  their  implementation.  For  example,  58%  of  the 
respondents  reporting  receiving  adequate  resources  for  risk  management  likewise  report 
using  qualitative  risk  techniques  during  project  selection  “Almost  Always,”  whereas  only 
10%  of  the  respondents  “Rarely”  receiving  such  support  also  report  “Almost  Always” 
using  risk  tools  during  project  selection. 

3 .  The  more  that  formal  risk  planning  practices  are  implemented,  the  more  rigorous 
is  risk  monitoring.  Also,  fewer  workarounds  are  reported.  For  example,  43%  of  the 
respondents  reporting  use  of  qualitative  risk  techniques  during  project  selection  also 
report  conducting  risk  reviews  “Almost  Always,”  whereas  only  10%  of  those  respondents 
reporting  “Rarely”  using  such  tools  also  report  “Almost  Always”  conducting  risk 
reviews. 

4.  The  more  that  project  success  is  reported,  the  stronger  senior  management  support 
for  formal  risk  planning  efforts,  their  subsequent  actual  practice,  and  regular  risk 
monitoring  are  also  reported.  For  example,  53%  of  the  respondents  who  report  their 
projects  conduct  risk  reviews  “Almost  Always”  also  report  completing  projects  on  time 
“Almost  Always,”  whereas  only  15%  of  the  respondents  who  report  “Rarely”  conducting 
risk  reviews  also  report  completing  projects  on  time  “Almost  Always.” 

In  conclusion,  there  is  a  statistically  significant  relationship  between  senior  management 
support  for  project  risk  management  and  the  presence  of  a  project  risk  management 
process  and  between  each  of  these  variables  and  reported  project  success.  This 
methodology  was  not  used  here  since  it  is  more  focused  on  the  value  of  risk  management 
versus  its  utility  in  program  selection. 

5  Excerpt  from  the  paper  by  the  Association  of  Reported  Project  Risk  Management  Practices  and  Project 
Success(published  in  2005),  from  Project  Management  Association  Finland,  Elisantie  16  02970  Espoo 
Finland 
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Another  methodology  provides  a  sample  risk  based  decisionmaking  graph.  6  Prior  to  the 
attack  on  the  USS  Cole  and  the  events  of  September  11,  2001  (9/11),  the  U.S.  Navy 
(Navy)  was  developing  antiterrorism  (AT)  programs  to  address  potential  terrorist  attacks 
in  a  resource-constrained  environment.  Amidst  unprecedented  tragedy  associated  with 
these  and  other  terrorist  strikes  against  U.S  interests,  both  domestic  and  abroad, 
government  agencies,  the  armed  forces,  and  private  companies  have  rapidly  executed  a 
vast  array  of  security  improvements  to  ensure  mission  capability  and  protect  personnel, 
critical  facilities,  transportation  systems,  and  other  infrastructure.  The  Navy  has  allocated 
resources  to  local  commanders  to  mitigate  these  identified  force  protection  challenges; 
however,  development  of  Navy -wide  investment  strategies  based  on  assessment  of  threat 
or  vulnerabilities  proved  programmatically  elusive,  thus  creating  the  need  for  a  different 
investment  protocol.  Given  the  challenges  of  allocating  limited  resources  among  many 
AT  capabilities,  each  with  demonstrated  abilities  to  reduce  the  vulnerability  of 
individuals  and  property  to  terrorist  acts,  the  Navy  is  incorporating  risk-based 
decisionmaking  (RBDM)  processes  (i.e.,  decision  processes  that  are  repeatable, 
consistent,  and  defendable)  into  its  resource  allocation  model.  This  risk  based 
decisionmaking  process  was  not  used  here  since  it  was  more  focused  on  operations 
versus  technology  program  selection. 

Another  approach  is  the  Science  &  Technology  Risk  Model  developed  by  the 
Homeland  Security  Institute.7  It  is  a  model  that  compares  all  S&T  capability  gaps  and 
programs.  It  is  based  on  three  elements:  attack/hazard  scenarios,  risk  calculations,  and 
mapping  gaps  and  programs  to  scenarios.  This  model  will  not  be  discussed  here  due  to 
the  sensitivity  of  the  dissemination  of  such  infonnation.  Based  on  the  review  of  these 
methodologies,  the  team  believes  that  the  current  figure  of  merit  discussed  in  this  paper  is 
more  relevant  to  the  decisionmaking  process  within  the  DHS  S&T  organization.  It  is 
based  on  decades  of  experience  by  the  study  team  in  laboratory  and  S&T  program 
management. 


6  Applying  Risk-Based  Decision-Making  Methods  and  Tools  to  U.S.  Navy  Antiterrorism  Capabilities,  by 
Charles  Mitchell  and  Chris  Decker.  Homeland  Defense  Journal,  February  2004. 

7  S&T  Risk  Model  for  Science  and  Technology  Planning  and  Resource  Allocation,  Volume  I-  Final  Report, 
26  August  2008,  Homeland  Security  institute,  Arlington,  VA. 
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Discussion-Risk  Areas  and  Impact  in  Decisionmaking 

There  are  many  methodologies  for  risk  informed  decisionmaking  as  outlined  in  the 
previous  section.  The  panel  decided  to  choose  one  particular  methodology  that  was 
developed  with  the  first  author  being  part  of  a  Defense  Science  Board  Study8.  The  Board 
adopted  a  figure  of  merit  technique  for  decisionmaking  in  the  DOD  technology  base.  This 
methodology,  balancing  opportunity  with  risk,  was  used  to  identify  the  top  17 
technologies.  The  same  concept  is  used  in  this  paper  with  different  elements  of 
opportunity  and  risk.  A  list  of  the  major  risk  areas  in  S&T  decisionmaking,  including 
both  technical  and  political  factors  was  produced.  Using  this  list,  the  team  created  logical 
sub-elements  for  each  risk  element  in  order  to  break  down  the  needs  in  greater  detail. 
Technical  risk  areas  included  skilled  technical  management;  S&T  workforce;  odds  of 
success;  facilities  and  equipment;  timeliness;  and  champion.  Political  risks  included 
earmarks;  Congressional  direction;  micromanagement  by  Congress  or  headquarters;  and 
champion. 

Additional  risk  areas  included  transition  risks;  financial  risks;  and  institutional  risks. 
Transition  risks  included  items  such  as  customer  pull,  trust  issues,  champion,  and 
timeliness.  Financial  risks  included  concerns  about  adequate  funding  for  the  effort, 
stability  of  funds,  and  timeliness.  Institutional  risks  included  relevance  of  mission, 
leveraging,  social  acceptability,  balance  of  programs  (near  term  vs.  short  term),  legal 
issues,  and  organizational  constraints. 

The  team  prioritized  each  of  the  sub-elements  using  a  “voting”  technique.  It  used  the 
“1/2  +1”  technique,  wherein  the  total  number  of  items  to  be  prioritized  is  divided  by  two, 
and  then  one  is  added  to  that  number.  There  were  eleven  items  to  consider,  so  that  1/2+  1 
equaled  6.5  which  was  rounded  off  to  7.  Each  team  member  then  assigned  the  number  7 
to  what  he  considered  the  most  important  sub-element,  the  number  6  to  the  next  most 
important  sub-element,  the  number  5  to  the  third  most  important  sub-element,  and  so 
forth.  The  total  votes  per  sub-element  were  tallied  and  the  process  resulted  in  the  risks 
being  prioritized  as  listed  below  -  with  1  being  the  element  providing  the  most  risk. 

1.  Workforce:  in-house  and  outside  capability 

2.  Customer:  lack  of  customer  support. 

3.  Adequacy  of  funding. 

4.  Mission  relevance:  part  of  core  mission  of  organization. 

5.  Odds  of  success:  technical. 

6.  Leveraging:  utilizing  resources  of  other  organizations. 

7.  Champion:  chain  of  command/Congress  willing  to  support  effort. 

8.  Balance  of  program:  long  term  vs.  short-term. 

9.  Timeliness:  may  not  meet  the  needs  of  the  day. 

10.  Facilities:  existence  and  accessibility. 

1 1 .  Social  acceptance/legality:  lack  of  public  acceptance. 


Report  of  the  Defense  Science  Board  1981  Summer  Study  Panel  on  the  Technology  Base,  November 
1981,  Office  of  the  Undersecretary  of  Defense  for  Research  and  Engineering,  Washington,  DC 
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It  should  be  noted  that  the  prioritization  of  the  risk  areas  is  that  of  the  team  members  and 
not  necessarily  DHS  priorities.  DHS  priorities  would  be  different,  especially  when  it 
came  to  issues  dealing  with  the  public  and  domestic  laws  and  regulations.  Following  risk 
prioritization,  the  team  listed  the  key  elements  of  impact,  utilizing  the  same  technique 
used  to  calculate  and  prioritize  the  risk  areas.  The  list  included,  from  most  to  least 
impact:  mission;  pervasiveness;  knowledge  base;  need  to  avoid  technical  surprises; 
leadership;  long-lasting  effects;  utility  of  work  to  others;  commercial  off-the  -shelf 
(COTS);  affordability;  and  convenience/acceptance.  The  risk  to  the  American  people  has 
many  dimensions.  For  example,  there  is  the  risk  that  a  technical  proposal  will  be 
unsuccessful,  perhaps  because  it  is  a  bad  idea,  or  is  premature,  or  the  available  funding  is 
inadequate.  The  risk  is  that  time  and  money  will  be  wasted.  There  is  a  risk  that  not 
funding  something  denies  the  development  of  a  capability  that  could  have  been  critically 
important,  hence,  failure  to  develop  costs  lives.  There  is  the  risk  of  developing  something 
that  creates  more  serious  problems  than  it  solves  and  itself  becomes  a  threat  to  the 
American  people  (examples  here  might  be  in  the  area  of  civil  liberties,  or  a  biology 
undertaking  that  backfires).  There  is  the  risk  of  spending  money  on  things  or  areas  that 
are  actually  not  risks  or  serious  threats.  There  is  the  perception  of  risk  and  the  actual  risk, 
e.g.,  we  perceive  that  shark  bites  are  common,  but  statistics  say  they  are  not.  Thus,  other 
than  high,  medium,  or  low  it  may  be  hard  to  come  up  with  meaningful  measures  of  risk. 

A  brief  description  of  each  of  the  impact  elements  follows.  Mission  value  involves  a 
major  contribution  to  mission  advancement.  Pervasiveness  involves  a  contribution  of 
some  value  to  a  wide  variety  of  missions.  Knowledge  base  means  contribution  to  new 
understanding,  as  well  as  the  development  of  a  quality  workforce.  Avoiding  technology 
surprises  means  knowing  the  technical  advances  others  are  making.  Leadership  implies 
international  expertise  in  the  area.  Duration  of  impact  implies  long-lasting  utilization. 
Utility  of  work  implies  for  DHS  utilization  at  state  and  local  levels.  Leveraging  COTS 
means  utilizing  commercial  off-the-shelf  equipment  whenever  possible.  Affordability 
means  cost  of  development,  implementation,  and  maintenance.  Convenience/acceptance 
means  minimal  public  disruption. 

The  team  next  demonstrated  how  a  figure  of  merit  for  a  selected  technology  area  can 
be  determined,  by  contrasting  risk  versus  impact  using  the  risk  elements  and  impact 
elements  discussed  above.  The  team  selected  three  test  cases:  for  each  test  case,  the  team 
assigned  a  9,  4,  or  1  to  each  risk  element  and  each  impact,  tallied  the  results,  and  plotted 
the  results  on  a  risk  vs.  impact  chart  (see  Figure  1).  Nine  was  considered  high,  four  was 
considered  moderate  and  one  was  considered  low.  The  figure  of  merit  was  calculated  as 
the  sum  of  impact  divided  by  the  sum  of  risk.  Values  were  determined  by  quantitative 
data  when  available,  as  well  as  by  qualitative  judgments.  For  this  activity,  the  team  used 
the  members’  consensus  to  arrive  at  the  nine,  four  or  one  result.  More  sophistication 
would  include  gathering  data  for  each  element,  thereby  providing  a  more  quantitative 
approach. 

The  team  used  three  sample  programs  to  demonstrate  the  methodology.  The  programs 
included  the  Army  Electric  Gun;  composites  for  the  Air  Force  B-2  bomber;  and  a  DHS 
High  Priority  Technology  Need.  In  the  DHS  case,  there  was  a  need  to  develop  the 
technology  to  enable  the  detection  of  person-borne  concealed  explosive  threats  in  various 
high-throughput  venues  at  standoff  distances. 
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Conceptualizing  Metrics 

A  chart  to  conceptualize  metrics  was  developed  that  plots  projects  against  a  risk 
(vertical)  and  an  impact  (horizontal)  axis.  The  best-case  scenario  would  be  a  project  that 
would  have  greatest  impact  with  minimal  risk.  This  approach  made  an  unambiguous 
distinction  between  a  successful  program,  the  B-2  bomber;  an  extremely  important 
program  dealing  with  IED  detection;  and  a  questionable  investment  opportunity,  the 
Army  electric  gun.  The  team  proposed  that  a  possible  approach  for  DHS  to  use  is  to  put 
all  of  its  programs  through  such  an  evaluation.  The  funds  supporting  those  programs  that 
are  high-risk  but  low  impact  could  become  available  to  reinvest  in  those  that  are  high 
impact/low  or  moderate  risk.  The  programs  that  have  the  same  impact-to-risk  ratio  could 
be  used  to  insure  that  a  proper  balance  is  achieved  between  near-  and  far-term 
investments;  in  other  words,  there  should  always  be  some  high-impact/high-risk 
programs  that  push  the  S&T  envelope  and  act  as  catalysts  for  paradigm  shifts  in  strategic 
and  technological  thinking.  Continuing  the  low-impact/low-risk  programs  should  be 
questioned. 


The  Risk  Management  and  Decision  Processes  Center  of  the  Wharton  School  of  the 
University  of  Pennsylvania  was  consulted  regarding  the  approach  adopted  in  this  study. 
The  Center  has  over  25  years  of  experience  in  various  aspects  of  risk  management 
research  for  both  the  public  and  private  sectors.  The  Center  agreed  with  the  approach 
taken  by  the  study  team  but  also  recommended  extending  the  figure  of  merit  over  time  to 
include  initial,  midterm,  and  final  term.  In  addition,  they  suggested  examining  the 
interdependency  of  the  programs  after  selection  and  addressing  any  concerns  raised  by 
such  examination. 
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Decisionmaking  Process  for  Program  Evaluation 

The  CTNSP  team  developed  a  flow  chart  for  the  decisionmaking  process  that  could 
be  used  by  evaluators  to  walk  through  the  risk  vs.  impact  evaluation  (see  Figure  2.)  It 
starts  with  existing  program  needs/requirements  or  new  ideas  that  have  to  be 
implemented.  These  go  through  the  impact  vs.  risk  assessment  process  detailed  earlier 
resulting  in  a  figure  of  merit  (FOM).  If  the  FOM  is  less  than  a  desired  threshold  x,  the 
leadership  can  still  go  forward  with  the  program  by  exercising  executive  privilege.  If  it  is 
“in  lane,”  then  it  should  be  accommodated.  By  “in-lane”  is  meant  that  the  program 
evaluators  determined  the  programs  fit  into  the  mainline  activities  of  the  DHS  and  have 
had  some  level  of  stable  funding  over  the  past  several  years,  with  that  level  projected  to 
remain  constant.  If  the  program  is  not  in  lane,  but  the  resulting  FOM  is  greater  than  the 
threshold  x,  then  it  goes  to  the  next  step  of  institutional  advocacy  process,  where  it  is 
vetted  against  funding  and  prioritizated  needs.  If  the  program/new  idea  falls  within  an 
existing  lane,  it  is  then  absorbed.  If  a  new  lane  has  to  be  created  for  the  program,  the 
laboratory  or  the  institution  is  either  appropriated  new  funding  for  it  or  it  absorbs  the 
costs.  If  the  laboratory/institution  has  to  absorb  the  cost  of  the  new  idea/program,  it 
would  most  likely  readjust  its  existing  programs  and  evaluate  them  against  prioritized 
needs.  Then,  funding  adjustments  would  be  made  on  existing  programs  to  support  the 
new  program.  This  decision  process  involving  selection  and  funding  can  be  applied  to  all 
new  ideas  and  to  existing  programs. 
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Decisionmaking  and  Program  Management 

During  the  course  of  the  preceding  evaluations,  and  in  the  day-to-day  operations  of  a 
laboratory,  the  CTNSP  team  members  suggested  that  there  are  at  least  five  different 
styles  of  decisionmaking  that  can  be  used  by  senior  leaders:  command  and  control 
(follow  orders  as  in  military);  consensus  building  (workable,  requires  knowledgeable 
leader,  takes  time  and  is  actually  harder,  provides  the  best  answer,  but  could  lead  to  less 
risk  taking);  self  policing  (requires  knowledgeable  leader  in  technologies,  uses  peer  group 
working  for  the  leader  to  evaluate  technologies);  intuitive  (using  intuition  after  reviewing 
inputs  from  organizations);  and  distributive  (leader  not  knowledgeable  in  technologies 
but  good  in  leadership  skills  and  management).  The  CTNSP  team  recommends  that 
command/control  and  distributive  styles  not  be  used  by  the  DHS.  The  team  did  favor  the 
consensus  building  approach,  but  cautioned  that  it  could  be  time  consuming.  Nonetheless, 
it  usually  leads  to  the  “best”  answer,  the  one  that  takes  into  account  many  diverse  views. 
Self  policing  is  effective  if  the  organization  is  well  populated  with  strong  technical  peers. 
The  intuitive  approach  should  be  limited  to  whenever  all  other  techniques  fail,  since  it 
relies  on  the  “gut  feel”  of  a  high-quality  leader  who  is  well-respected. 

While  the  team  did  not  discuss  in  detail  the  differences  between  leaders  and 
managers,  the  members  agreed  it  was  important  to  recognize  they  are  different  concepts. 
Warren  Bennis,9  a  noted  authority  on  this  subject,  identified  several  differences: 

•  Managers  administer,  leaders  innovate 

•  Managers  ask  how  and  when,  leaders  ask  what  and  why 

•  Managers  focus  on  systems,  leaders  focus  on  people 

•  Managers  do  things  right,  leaders  do  the  right  things 

•  Managers  maintain,  leaders  develop 

•  Managers  rely  on  control,  leaders  inspire  trust 

•  Managers  have  a  short-term  perspective,  leaders  have  a  longer-tenn  perspective 

•  Managers  accept  the  status-quo,  leaders  challenge  the  status-quo 

•  Managers  have  an  eye  on  the  bottom  line,  leaders  have  an  eye  on  the  horizon 

•  Managers  imitate,  leaders  originate 

•  Managers  emulate  the  classic  good  soldier,  leaders  are  their  own  person 

•  Managers  copy,  leaders  show  originality 

Most  organizations  are  usually  populated  with  good  managers  and  fewer  good 
leaders.  However,  balance  between  leadership  and  management  is  key  to  the  scientific 
organization’s  success.  The  team  reiterated  that  fortunately,  leadership  can  be  an  acquired 
quality  and  can  be  learned  with  enough  dedication  and  hard  work.  However,  too  little  of 
this  key  training  is  actually  done.  Based  on  the  personal  experience  of  team  members, 
most  directors  of  research  and  development  organizations  have  both  good  leadership  and 
managerial  skills. 


9  http://www.phd.antioch.edu/Pages/APhDWeb_Program/bennis.  Dr.  Warren  Bennis  (Antioch  College,  Class  of 
1951)  is  one  of  the  nation's  foremost  authorities  on  organizational  development,  leadership  and  change.  He  is  currently 
a  distinguished  member  of  the  Antioch  College  Ph.D.  Program's  Panel  of  Visitors,  and  was  a  member  of  Antioch 
University's  Board  of  Trustees  (1968-71). 
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Conclusions 

A  risk  versus  impact  analysis  tool  for  decisionmaking  was  developed.  This  tool 
utilized  technical,  political,  transition,  financial,  and  institutional  risk  areas  that  were 
prioritized.  The  impact  areas  included  mission,  pervasiveness,  knowledge  base,  avoiding 
surprise,  leadership,  longevity,  utility  to  others,  commercial  off-the-shelf  (COTS), 
affordability,  and  convenience/acceptance  to  the  American  public.  The  impact  areas  were 
also  prioritized.  A  figure  of  merit,  which  is  the  sum  of  the  impact  areas  divided  by  the 
sum  of  the  risk  areas,  was  generated.  This  figure  of  merit  was  applied  to  three  examples: 
Army  electric  gun,  composites  for  AF  B-2,  and  the  DHS  high  priority  technology  need  of 
detecting  person-borne  IEDs  from  a  standoff  distance.  The  team  then  examined  the  flow 
process  for  potential  program  selection  and  funding.  The  styles  of  decisionmaking  within 
DOD  Laboratory  management  were  then  considered  and  the  differences  between 
leadership  and  management  were  highlighted.  The  team  believes  that  the  tools  provided 
in  this  paper  will  be  of  value  to  the  DHS  S&T  leadership  in  the  process  of  program 
selection  and  ultimate  funding.  The  next  step  is  to  apply  these  tools  to  other  high-priority 
technology  needs  within  the  various  divisions  of  the  DHS  S&T  Directorate.  Of  course, 
these  risk-informed  decisionmaking  tools  can  be  used  by  any  organization  for  analytical 
S&T  program  decisionmaking. 
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Appendix.  Performance  Evaluation  and  Metrics  for 
Program  Selection,  Army  Research  Laboratory 


Army  Research  Lab  introduced  its  management  innovation,  which  was  a  methodology  to 
evaluate  the  health  and  performance  of  the  laboratory.  Research  &  Development 
performance  evaluation  had  been  a  problem  for  the  Anny  that  has  eluded  a  satisfactory 
solution  for  many  decades. 

ARL  developed  a  Performance  Evaluation  Construct.  It  was  a  semi-quantitative  approach 
that  required  the  Lab  Director  to  take  a  broad  view  of  many  different  factors,  some 
numeric  and  some  descriptive,  and  then  to  personally  integrate  them  into  a  picture  that  he 
can  present  to  a  variety  of  audiences  (including  stakeholders)  in  whatever  format  is 
appropriate  at  the  moment.  Other  evaluation  areas  were  peer  review,  customer  evaluation, 
relevance,  productivity  and  quality.  The  interrelationship  of  all  six  constructs  was  then 
plotted  on  the  following  chart: 


Relationship  of  three  pillars  of  ARL  Performance  Evaluation  Construct 
to  principal  areas  of  interest 


METHOD 

GOAL 

Relevance  Productivity  Quality 

Peer  review 

o  o  • 

Metrics 

o  o  o 

Customer  evaluati  on 

•  •  o 

Very  useful 

Samev/hat  useful  1,  )  Less  useful 

Figure  3.  Performance  Evaluation  Construct 
Peer  Review 

To  better  utilize  the  concept  of  peer  review,  ARL  contracted  with  the  National  Research 
Council  of  the  National  Academies  of  Science  and  Engineering  to  assemble  an  ARL 
Technical  Assessment  Board  (TAB).  The  TAB  consists  of  individuals  of  international 
reputation.  Under  this  board  are  six  panels  of  8  to  10  people  each,  of  high  reputation  in 
the  international  community.  These  panels  review  each  of  the  ARL  primary  mission 
areas.  The  Board,  with  its  panels,  provides  an  appraisal  of  the  scientific  and  technical 
efforts  of  ARL.  It  is  specifically  enjoined  from  making  judgments  on  the  programmatic 
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structure  of  ARL  work,  partly  because  it  is  not  equipped  to  perform  this  function,  and 
partly  because  there  are  numerous  other  channels  through  which  ARL  receives  such 
guidance. 

Because  of  the  size  and  diversity  of  the  ARL  technical  program,  the  panels  review  the 
total  program  in  depth  over  a  three-year  period,  reporting  annually  to  the  TAB  on  the  one 
third  of  the  program  reviewed.  The  panels  produce  a  descriptive  assessment  in  the  form 
of  a  written  report  published  by  the  National  Research  Council.  The  TAB  also  meets 
annually  with  the  Director  to  provide  an  infonnal  report  of  its  findings  before  the 
publication  of  the  report,  and  to  receive  guidance  on  the  Director’s  desires  for  special 
areas  of  emphasis  in  the  next  year’s  review. 

Metrics 

ARL  assembled  a  set  of  about  60  metrics  for  consideration.  They  were  broken  into  5 
major  categories: 

Preeminent  in  key  areas  of  science 
Staff  widely  recognized  as  outstanding 
Miscellaneous 

Seen  by  Anny  users  as  essential  to  their  mission 
Intellectual  crossroads  for  the  technical  community 

Of  the  60  metrics  proposed,  17  were  selected: 

•  %  of  top  tasks  met; 

•  %  of  Science  and  Technology  Objectives  met; 

•  #  of  refereed  papers/proceedings  leaving  tracks; 

•  #  of  ARL  technical  reports  leaving  tracks; 

•  %  of  Ph.D.s  on  the  staff; 

•  #  of  employees  on  long-tenn  training; 

•  #  of  employees  on  academic  training; 

•  total  semester  credit-hours  completed  by  Ph.D.  candidates; 

•  indirect  overhead  in  ($M); 

•  ratings  form  customer  surveys-  technical  planning  annexes  and  reimbursable 

customers; 

•  reimbursable  customer  orders  (in  $M); 

•  total  #  of  guest  researchers  out  of  ARL; 

•  total  work-year  equivalents  for  guest  researchers  out  of  the  ARL; 

•  total  #  of  guest  researchers  into  ARL(#  of  post-docs); 

•  total  work-year  equivalents  of  guest  researchers  into  ARL. 
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Lessons  learned 

Lessons  learned  from  the  ARL  performance  evaluation  were: 

The  survey  process  for  customer  feedback  is  more  applicable  to  the  applied  research 
that  ARL  does  than  to  the  more  basic,  opportunity-driven  research. 

Peer  review  is  more  applicable  to  the  ARL  basic  research  work 

Metrics  are  applicable  with  varying  emphasis,  depending  on  the  metric  and  when  it  is 

applied. 

This  means  that  the  three  pillars  of  the  perfonnance  evaluation  construct  are  given 
different  weights  at  different  points  in  the  R&  D  life  cycle.  The  use  of  the  metrics 
presents  some  difficulties.  Good  management  practice  suggests  that  only  a  few  key 
metrics  will  be  useful  in  evaluating  perfonnance.  The  metrics  developed  above  are 
mostly  measures  of  the  research  work  while  in  progress,  not  of  the  ultimate  use  of  the 
results.  The  laboratory  has  little  control  as  to  whether  or  not  the  parent  organization  is 
able  to  make  good  use  of  the  research  findings  and,  furthennore,  the  ultimate  outcome  of 
the  laboratory  work  may  be  several  years  in  the  future.  In  the  Anny,  it  may  be  one  or  two 
decades  between  the  laboratory  demonstration  of  a  new  concept  and  its  fielding  in  a  war 
fighting  system.  ARL  has  concluded  that  there  simply  are  no  meaningful  outcome 
metrics,  in  the  sense  that  a  manufacturing  or  service  company  would  use  them.  ARL  has 
concluded  that  there  is  no  satisfactory  approach  to  predicting  the  utility  of  laboratory 
results  many  years  in  the  future.  This  has  been  confirmed  in  extensive  discussions  with 
the  senior  leaderships  of  leading  American  technology  companies. 
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